{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "21cfea06-af98-496a-b13b-106c335a2e65",
   "metadata": {},
   "source": [
    "# Understand Text Similarity Space\n",
    "\n",
    "`TextSimilaritySpace` creates the opportunity to embed text data to perform semantic search on top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1bf67328-efe5-4c88-9c36-ce2d4b20d89f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install superlinked==37.5.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "11664035-fff3-4c38-97f3-f2fbb0d46778",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from superlinked import framework as sl\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2468c61d-7e23-4b3e-a31a-04a9ed7df676",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Paragraph(sl.Schema):\n",
    "    id: sl.IdField\n",
    "    body: sl.String\n",
    "\n",
    "\n",
    "paragraph = Paragraph()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ff0c879-a280-4e95-ada0-4b0395199725",
   "metadata": {},
   "source": [
    "## Choosing your text embedding model\n",
    "\n",
    "Currently we support [Sentence Transformers](https://www.sbert.net/docs/pretrained_models.html#model-overview) models for embedding textual data, as they are tailor-made for information retrieval on sentence long text.\n",
    "\n",
    "Supply a [huggingface identifier](https://huggingface.co/sentence-transformers) of your choosing, and set the relevant SchemaField containing your text inputs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2df24eaf-f9b8-404b-8b7f-0a9d2c2284df",
   "metadata": {},
   "outputs": [],
   "source": [
    "body_space = sl.TextSimilaritySpace(text=paragraph.body, model=\"sentence-transformers/all-mpnet-base-v2\")\n",
    "paragraph_index = sl.Index(body_space)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4aed826a-470e-4e53-8f2a-e8e529373b61",
   "metadata": {},
   "outputs": [],
   "source": [
    "source: sl.InMemorySource = sl.InMemorySource(paragraph)\n",
    "executor = sl.InMemoryExecutor(sources=[source], indices=[paragraph_index])\n",
    "app = executor.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "61d76e31-72f4-4404-8cc1-31c08d08d978",
   "metadata": {},
   "outputs": [],
   "source": [
    "source.put(\n",
    "    [\n",
    "        {\"id\": \"paragraph-1\", \"body\": \"Glorious animals live in the wilderness.\"},\n",
    "        {\n",
    "            \"id\": \"paragraph-2\",\n",
    "            \"body\": \"Growing computation power enables advancements in AI.\",\n",
    "        },\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "198f0bce-5d7c-412b-a989-d0ab60982770",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = sl.Query(paragraph_index).find(paragraph).similar(body_space, sl.Param(\"query_text\")).select_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "aaa75750-3b1c-4d68-af9c-133a249f6b0b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>body</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Growing computation power enables advancements in AI.</td>\n",
       "      <td>paragraph-2</td>\n",
       "      <td>0.462400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Glorious animals live in the wilderness.</td>\n",
       "      <td>paragraph-1</td>\n",
       "      <td>-0.040457</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    body           id  \\\n",
       "0  Growing computation power enables advancements in AI.  paragraph-2   \n",
       "1               Glorious animals live in the wilderness.  paragraph-1   \n",
       "\n",
       "   similarity_score  \n",
       "0          0.462400  \n",
       "1         -0.040457  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(\n",
    "    query,\n",
    "    query_text=\"What makes the AI industry go forward?\",\n",
    ")\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
